dsh-skill-eval: An Evaluation Plugin Using LLM Judge to Quantify DSH Skill Trigger Reliability

# Introduction In DSH (DeepSeek Harness), whether a skill is utilized depends largely on its description: the model determines whether to invoke a specific skill based on the description provided in the skill catalog prompt. If the description is too narrow, matching queries will fail to trigger the skill (under-triggering); if it is too wide, irrelevant queries will also be triggered (over-triggering). Addressing these issues typically requires manual testing with a few queries based on intuition, resulting in outcomes that are both unreproducible and difficult to compare. `dsh-skill-eval` transforms this into a reproducible evaluation: it uses an LLM judge to fully replicate the official skill catalog prompt, determines whether each test query should trigger the target skill, and aggregates the results into metrics.